Item Response Theory Parameter Estimates Research Articles

Application of Sampling Variance of Item Response Theory Parameter Estimates in Detecting Outliers in Common Item Equating.

In common item equating, the existence of item outliers may impact the accuracy of equating results and bring significant ramifications to the validity of test score interpretations. Therefore, common item equating should involve a screening process to flag outlying items and exclude them from the common item set before equating is conducted. The current simulation study demonstrated that the sampling variance associated with the item response theory (IRT) item parameter estimates can help detect outliers in the common items under the 2-PL and 3-PL IRT models. The results showed the proposed sampling variance statistic (SV) outperformed the traditional displacement method with cutoff values of 0.3 and 0.5 along a variety of evaluation criteria. Based on the favorable results, item outlier detection statistics based on estimated sampling variability warrant further consideration in both research and practice.

An Investigation of Age-Related Differential Item Functioning in the EQ-5D-5L Using Item Response Theory and Logistic Regression

ObjectivesIn economic evaluations, quality of life is measured using patient-reported outcome measures (PROMs), such as the EQ-5D-5L. A key assumption for the validity of PROMs data is measurement invariance, which requires that PROM items and response options are interpreted the same across respondents. If measurement invariance is violated, PROMs exhibit differential item functioning (DIF), whereby individuals from different groups with the same underlying health respond differently, potentially biasing scores. One important group of healthcare consumers who have been shown to have different views or priorities over health is older adults. This study investigates age-related DIF in the EQ-5D-5L using item response theory (IRT) and ordinal logistic regression approaches. MethodsMultiple-group IRT models were used to investigate DIF, by assessing whether older adults aged 65+ years and younger adults aged 18 to 64 years with the same underlying health had different IRT parameter estimates and expected item and EQ-5D-5L level sum scores. Ordinal logistic regression was also used to examine whether DIF resulted in meaningful differences in expected EQ level sum scores. Effect sizes examined whether DIF indicated meaningful score differences. ResultsThe anxiety/depression item exhibited meaningful DIF in both approaches, with older adults less likely to report problems. Pain/discomfort and mobility exhibited DIF to a lesser extent. ConclusionsWhen using the EQ-5D-5L to evaluate interventions and make resource allocation decisions, scoring bias due to DIF should be controlled for to prevent inefficient service provision, where the most cost-effective services are not provided, which could be detrimental to patients and the efficiency of health budgets.

Detecting Differential Item Functioning: Item Response Theory Methods Versus the Mantel-Haenszel Procedure

This Monte Carlo study assessed Type I error in differential item functioning analyses using Lord’s chi-square (LC), Likelihood Ratio Test (LRT), and Mantel-Haenszel (MH) procedure. Two research interests were investigated: item response theory (IRT) model specification in LC and the LRT and continuity correction in the MH procedure. This study enhances the literature by investigating LC and the LRT using correct and incorrect model-data fit and comparing those results to the MH procedure. There were three fixed factors (number of test items, IRT parameter estimation method, and item parameter equating) and four varied factors (IRT model used to generate data and fit the data, sample size, and impact). The findings suggested the MH procedure without the continuity correction is best based on Type I error rate.

Reliability of Differential Item Functioning in Alcohol Use Disorder: Bayesian Meta-Analysis of Criteria Discrimination Estimates.

Numerous studies leverage item response theory (IRT) methods to examine measurement characteristics of alcohol use disorder (AUD) diagnostic criteria. Less work has examined the consistency of AUD IRT parameter estimates, an essential step for establishing measurement invariance, making statements about symptom diagnosticity, and validating the theoretical construct. A Bayesian meta-analysis of IRT discrimination values for AUD criteria across 33 independent samples (Total N = 321,998) revealed that overall consistency of AUD criteria discriminations was low (generalized intraclass correlation range = .105-.249). However, specific study characteristics accounted for substantial variability, suggesting that the unreliability is partially systematic. We replicated evidence of differential item functioning (DIF) via established factors (e.g., age, gender), but the magnitudes were small compared with DIF associated with assessment instrument. These results offer practical recommendations regarding which instruments to use when specific AUD criteria are of interest and which criteria are most sensitive when comparing demographic groups.

The Uniform Prior for Bayesian Estimation of Ability in Item Response Theory Models

Item Response Theory (IRT) models traditionally assume a normal distribution for ability. Although normality is often a reasonable assumption for ability, it is rarely met for observed scores in educational and psychological measurement. Assumptions regarding ability distribution were previously shown to have an effect on IRT parameter estimation. In this study, the normal and uniform distribution prior assumptions for ability were compared for IRT parameter estimation when the actual distribution was either normal or uniform. A simulation study that included a short test with a small sample size and a long test with a large sample size was conducted for this purpose. The results suggested using a uniform distribution prior for ability to achieve more accurate estimates of the ability parameter in the 2PL and 3PL models when the true distribution of ability is not known. For the Rasch model, an explicit pattern that could be used to obtain more accurate item parameter estimates was not found.

Proceedings of Contributed Presentations 10th World Federation of Chiropractic/Association of Chiropractic Colleges Education Conference

The World Federation of Chiropractic and the Association of Chiropractic Colleges have hosted a biennial joint education conference for ten years with this being the second time that the proceedings were published in a peer-reviewed journal. The Journal of Chiropractic Education was delighted to sponsor the meeting by donating the work necessary to edit the proceedings for publication and post-production indexing, thereby making these proceedings available in indexing systems searched by academics at universities worldwide, including PubMed, Emerging Sources Citation Index, Scopus, the Index to Chiropractic Literature, the Cumulative Index to Nursing and Allied Health Literature, the Allied and Complementary Medicine Database, and MANTIS. Brighthall Inc, managed the call for papers, submissions, and administration of the peer-review process.The conference organizers recognized the international development of chiropractic training programs and the value of interactive formats and agreed to provide 4 different types of presentations for researchers to submit their work for peer review: platform presentation, poster presentation, innovation panels, and workshops.The call for papers was distributed in March 2018 and submissions were due May 1, 2018. All submissions were peer reviewed within a month by 45 independent reviewers from nearly every chiropractic training program around the world, represented by reviewers from 16 different countries. The peer-review committee is commended for exemplary work performed over a very short time. The peer reviewers included: Hasan Kerem Alptekin, MD, Bahçesehir Üniversity; Fernando R.M. Azevedo, DC, Universidade Anhembi Morumbi; Richard Brown, DC, LLM, World Federation of Chiropractic; David Byfield, DC, MPhil, Welsh Institute of Chiropractic; Robert Cooperstein, MA, DC, Palmer College of Chiropractic West; Barry Draper, DC, PhD, Central Queensland University; Ana Facchinato, DC, MHS, Southern California University of Health Sciences; Ricardo Fujikawa, MD, DC, Real Centro Univeristario Escorial Maria Cristina; Rosemary Giuriato, DC, DO, Macquarie University; Dominic Giuliano, DC, Canadian Memorial Chiropractic College; Chris Good, DC, MAEd, University of Bridgeport College of Chiropractic; Bart Green, DC, MSEd, PhD, Stanford Health Care, National University of Health Sciences; Julie-Marthe Grenier, DC, Université du Québec à Trois-Rivières; Navine Haworth, DC, PhD, Victoria University; Michael Haneline, DC, MPH, International Medical University; Xiaohua He, MD, MS, Palmer College of Chiropractic Florida; Sean Herrin, DC, University of Western States; Kelly Holt, BSc, PhD, New Zealand College of Chiropractic; Adrian Hunnisett, PhD, MPhil, McTimoney College of Chiropractic; Claire Johnson, DC, MSEd, PhD, National University of Health Sciences; HanSuk Jung, DC, PhD, HanSeo University; Martha Kaeser, DC, Med, Logan University; Amilliah Kenya, DC, MS, Sherman College of Chiropractic; Charmaine Korporaal, MTech, Durban University of Technology; Arnaud Lardon, DC, PhD, Institut Franco-Européen de Chiropratique; Henrik Hein Lauridsen, DC, MSc, PhD, Syddansk Universitet Odense; Deborah Barr, MS, SCD, New York Chiropractic College; Clay McDonald, DC, MBA, JD, Logan University; John Mrozek, DC, Med, Texas Chiropractic College; Daniel Mühlemann, DC, University of Zurich; David Newell, PhD, AECC University College; Per J. Palmgren, RC, MMedEd, PhD, Karolinska Institutet; Lisa Rubin, PhD, Life University; Eric Russell, DC, Parker University; Thiana Paula Schmidt dos Santos, DC, Centro Universitario Feevale; Monica Smith, DC, PhD, Life Chiropractic College West; Christopher Smoley, DC, D'Youville College; Michael Tunning, DC, ATC, MS, Palmer College of Chiropractic; Adrian Wenban, DC, MMedSc, Barcelona College of Chiropractic; Stephney Whillier, BSc, BEd, PhD, Macquarie University; Michael Wiles, DC, MEd, MS, Keiser University; Chris Yelverton, MTech, University of Johannesburg; Kenneth Young, DC, MAppSc, Murdoch University.Tobias Barnard, Atheesha Singh, Shelley Blundell, Christopher YelvertonProper hand hygiene has been described as an important barrier in the defense against hospital acquired infections, along with the spread of antimicrobial resistance in the bacteria that cause these infections. Research has shown that although healthcare workers understand the principles and importance of hand hygiene, they do not always practice proper hand hygiene. The aim of this pre-post intervention study was to monitor the changes in the bacterial population on the hands of chiropractic students at a Chiropractic Clinic pre and post a hand hygiene education intervention. Sixty participants completed the World Health Organization hand hygiene survey to determine their hand hygiene knowledge, attitude and practices (KAP), and had their hands sampled to quantify the bacterial population present. The bacterial population was quantified using flow cytometry and reported as total, live and dead bacteria present. The survey answers showed that the participants reported knowledge, attitudes and practices did improve after the education intervention (presentations and posters in toilets and consultation rooms). The microbiological data showed that although there was a decrease in the total bacterial population after the intervention, the portion live cells increased after the intervention. Comparing the change in microbiological data with the change in KAP answers provided (pre- and post-intervention), showed that although the participants understood the importance of proper hand hygiene practices, it was not supported by the microbiological data. The data showed that the effectiveness of hand hygiene education need to be monitored using a microbiological method to properly combat the spread of infections. (This is a conference presentation abstract and not a full work that has been published.)Lydia Brodie, Pierre Côté, Christine Mechamalil, Sheilah Hogg-Johnson, Canadian Memorial Chiropractic College; Kathy Smith, Craig Jacobs, Linda CarrollObjective: To determine the one-week prevalence of neck pain and low back pain in this population. To determine whether neck and low back pain is correlated with psychological stress in undergraduate chiropractic students attending the Canadian Memorial Chiropractic College (CMCC). Methods: We conducted a cross-sectional study of the CMCC students during the fall semester 2017. We collected data using the CMCC Mental Health and Wellness Questionnaire, an electronic questionnaire designed for in-class administration. The questionnaire includes measures of neck and low back pain intensity (0-10 Numeric Pain Rating Scale) and psychological stress (DASS-21 scale - Depression, Anxiety and Stress Scale). The questionnaire also includes instruments to measure lifestyle factors. We measured the correlation between neck pain, low back pain and stress using Pearson's correlation. Results: We recruited 510 students, for a participation rate of 67%. The mean age of our sample was 24.6 years with 60% female. Overall, the one week prevalence of neck pain was 76.9%, 95% CI [73,81] and of low back pain 69.0%, 95% CI [65,73]. Stress was reported as normal in 63.3%, mild 14.9%, moderate 16.7%, severe 3.9% and extremely severe 1.2% over one week. We found that neck pain and stress (r=0.348) and low back pain and stress (r=0.259) were positively correlated. Conclusion: A significant number of CMCC students report experiencing neck pain, low back pain and symptoms of psychological stress. Understanding whether pain and psychological stress are associated may inform the management and prognosis of neck and back pain in chiropractic students. (This is a conference presentation abstract and not a full work that has been published.)Danny Clegg, David Byfield, Alister DuRosePsychomotor skills training is a key component of a Chiropractic education programme, and is considered an educational challenge. As well as individual proficiency issues, students often report difficulties with time-distribution of tutors, particularly when they are seeking reinforcement that they are making progress. Learning complex psychomotor skills requires time and repetition, therefore feedback is an important learning tool for students. In order to address these issues, a digital platform was sourced and adapted for the purpose of increasing student feedback through self-directed and directed mediums. VEO is a digital application which allows videos recorded by instructors and students to be uploaded from any device to a secure, dedicated server, for self-assessment and direct tutor feedback as required. VEO was introduced as part of the year 2 Chiropractic manual skills module during the 2017-18 academic year, with students voluntarily invited to participate as both part of, and an adjunct to, classroom learning. Standard setting of all VIVA examination assessment points was completed between the assessors prior to examination, and none of the assessors were aware of VEO engagement data. The initial cross-sectional study identified a significant positive correlation between the number of videos students engaged in and VIVA 1 performance (r=0.273, p=0.038). Follow-up analysis included performance for VIVA 1 and VIVA 2. Significant positive correlation was again identified between the number of videos students engaged in with both VIVA 2 (r=0.272, p=0.039), and overall VIVA outcomes (r=0.350, p=0.007). Further significant positive correlation was identified between overall VIVA outcomes and total number of videos uploaded directly by students for adjunctive feedback (r=0.340, p=0.009). This study indicates that employing VEO enhances manipulative psychomotor skill support and acquisition. (This is a conference presentation abstract and not a full work that has been published.)David Côté, Kathy Smith, Pierre CôtéObjective: To measure the one-week prevalence of moderate to extremely severe psychological distress (depression, anxiety and stress) and three-month prevalence of substance use (alcohol, tobacco, cannabis, amphetamines and hallucinogens) in chiropractic students enrolled at the Canadian Memorial Chiropractic College (CMCC). We also measured the association between psychological distress and substance use. Methods: We conducted the CMCC Mental Health and Wellness Cross-sectional Study in the Fall of 2017. Enrolled students at CMCC were eligible for participation. We measured depression, anxiety and stress using the Depression, Anxiety, Stress Scales-21 and substance using the WHO ASSIST questionnaire. We report the prevalence of psychological distress and substance use. We used multivariable logistic regression to measure the association between psychological distress and substance use. Results: The participation rate was 67% (510/766). The mean age was 24.6 years, and 60% were females. The one-week prevalence of moderate to extremely severe symptoms was 19.0% for depression, 32.6% for anxiety and 21.8% for stress. In the past three months, 91.0% reported using alcohol, 33.1% used cannabis, 18.0 % used tobacco, 5.1% used amphetamines and 4.5% used hallucinogens. The associations between substance use and mental health outcomes will be presented at the conference. Conclusions: A significant proportion of CMCC students report moderate to extremely severe symptoms of depression anxiety, stress. We fond that the consumption of alcohol, cannabis, and tobacco is high. Understanding the prevalence of psychological distress and substance use in chiropractic students is important for educators because they impact academic performance and attainment of future career goals. (This is a conference presentation abstract and not a full work that has been published.)Ian Coulter, Gery Ryan, Lea Xenakis, Lisa Kraus, Lara HiltonIn deconstructing the placebo scholars have elevated the importance of the context of the health encounter recognizing that the encounter may not simply be a non-specific effect. This study developed a rapid ethnographic observation method to study the context of the encounter. What kinds of contextual factors are patients exposed to during chiropractic encounters; how do you measure such contextual factors reliably via observation and/or patient and provider recall; how do you assess the degree to which contextual factors might vary within and across practice sites, providers, and individual patients. We defined healthcare encounters as what patients experience between when they arrive at a practice site to when they exit the building. Any given encounter can be described along 5 key logical dimensions: (1) where patients are (space), (2) with whom they interact (social), (3) what is communicated between them (communication), (4) what patients do or what is done to them (behavior), and (5) for how long and in what order 1-4 occur (time). This study was conducted in three phases. In Phase 1, we conducted focus groups with chiropractic patients to delineate these features of the health encounter thought to be important to patients. In Phase 2, we conducted a pilot rapid ethnographic observation study of a single chiropractic clinic to determine how to measure the elements of the encounter identified by the patient focus groups. In Phase 3, we conducted a national study in three states using a representative sample in each state. (This is a conference presentation abstract and not a full work that has been published.)Ian Douglass Coulter, Patricia HermanObjective: To expand the RAND/UCLA appropriateness of care methodology to include patient preferences and resource utilization, and the impact of care appropriateness on patient outcomes. Data Sources/Study Setting Primary data from expert panels, focus groups, chiropractors, chiropractic patients with chronic low back pain (CLBP) and chronic cervical pain (CCP), and from internet “workers” via crowdsourcing. Study setting is a cluster sample of 125 chiropractic clinics from six US regions. Study Design This multicomponent methods study includes analysis of longitudinal data on patient outcomes, preferences, CLBP and CCP symptoms and healthcare utilization. Data Collection/Extraction Methods: Data were collected bi-weekly on 2025 patients for 3 months via online surveys that included both new and legacy measures, including PROMIS and CAHPS. Principle Findings: Appropriateness panels generated ratings for 450 CLBP and 180 CCP indications which were then applied to patient charts. Data was collected from 2025 patients and from 3000 patient files. Conclusions: Patient-centered care is a significant policy initiative but translating it into policy that has been clinician and research-expert based, poses significant methodological issues. Nonetheless, we make the case that patient preferences, self-reported outcomes, and financial burden should be considered in the evaluation of the appropriateness of healthcare. (This is a conference presentation abstract and not a full work that has been published.)Owis Eilayyan, Aliki Thomas, Sara Ahmed, Anthony Tibbles, Craig Jacobs, Fadi Alzoubi, Andre BussieresIntroduction: Despite guidelines recommending clinicians use Self-Management Support (SMS), uptake is suboptimal. Previously identified barriers to using SMS among chiropractors, interns and patients informed the design of a knowledge translation (KT) intervention for use in chiropractic teaching clinics. Objective: To estimate the feasibility and potential effectiveness of a KT intervention to promote the use of SMS among chiropractors, interns and patients with spine pain compared to control. Methods: Pilot clinical trial across 4 outpatient-teaching clinics. Twenty Patient Management Teams (PMTs), each composed of 6-9 interns supervised by a clinician, were allocated to either the intervention (training workshop, webinar, e-educational module, and opinion leader) or wait-list. We assessed clinicians' and interns' SMS perceived importance, skills and confidence. Results: Sixteen (84%) clinicians and 39 (29%) interns agreed to participate. Clinicians (n=7 and n=9) and interns (n=17 and n=22) were allocated to the KT intervention and control groups respectively. Nearly all clinicians completed baseline and first follow-up surveys. 16 and 15 interns in the intervention and control group completed the baseline surveys respectively, while 11 and 10 interns completed second follow-up surveys. Preliminary estimates showed that intervention group clinicians had greater improvements in SMS perceived importance (mean change 0.24vs-0.02), skills (1.1vs0.43), and confidence (0.51vs0.35) compared to controls. Interns in both groups had mixed results. Conclusion: Preliminary results of this ongoing trial suggest that conducting a larger implementation trial in this setting is feasible. Theory-based tailored KT interventions may increase the likelihood of effective uptake and application of guideline recommendations within academic teaching institutions. (This is a conference presentation abstract and not a full work that has been published.)Karin Hammerich, Kent Stuber, Anser Abbas, Martin Harris, Sheilah Hogg-Johnson, Henrik Hein Lauridsen,Nadege Lemeunier, Michele MaiersBackground: A patient-centered approach is desirable in patient care. Attitudes of students towards patient-centred care in other health professions has been assessed; however, what chiropractic student attitudes are towards patient-centered care and variations between programs is unknown. Objective: To assess student attitudes towards patient-centred care among selected chiropractic programs worldwide. Methods: Students from seven chiropractic educational programs completed an online survey consisting of demographics and the Patient-Practitioner Orientation Scale (PPOS). PPOS assesses patient-centred attitudes towards the doctor-patient relationship. The PPOS provides scores between 1 and 6, (lower scores suggesting a more doctor-centred approach; higher scores, a more patient-centred approach). Results were analyzed descriptively and inferentially for overall, sharing and caring subscales. A general linear regression model identified factors associated with PPOS scores. Results: There were 1858 respondents (50.8% response rate). Student average age was 24.7 (range=17-58) years and 56.2% were female. The average overall PPOS score was 4.18 (SD=0.48), while the average sharing and caring subscale scores were 3.89 (SD=0.64) and 4.48 (SD=0.52), respectively. Gender, age, and program contributed small but significant differences in all PPOS scores. Year/semester of study within program typically did not significantly affect scores, neither did history of previous chiropractic care or having family members who are health professionals. Conclusion: We found small but significantly different PPOS scores between programs. Average scores tended to trend slightly lower than those reported in other health professions. Research exploring other factors, including curricular content and student beliefs, may help explain the observed student patient-centred attitudes. (This is a conference presentation abstract and not a full work that has been published.)Igor Himelfarb, Bruce ShottsThe assumption of local independence (LID) is violated when several items refer to the same vignette or items are related in other ways. The NBCE Written Clinical Competency Exam (Part III) consists of two sections, with a total of 110 multiple-choice questions and 10 case vignettes. The nature of the vignette items violates the assumption of LID for items nested within a vignette. We consider these items testlets. Due to the close relevance of within-testlet items, the LID assumption is violated, ignoring what often results in item parameter estimation bias (Wainer, Sheehan, and Wang, 2000) and overestimation of measurement reliability. (Sireci, Thissen, and Wainer, 1991) The objective of this paper is to propose a model for polytomouslyscored items with violation of LID. The model is based on the Testlet Response Theory (TRT) proposed by Wainer, Bradlow, and Wang (2007). The approach focuses on modeling LID caused by testlets; thus, allowing the estimation of the tests at item-level. In this paper, we propose a two-level unidimensional TRT estimation algorithm which takes advantage of their approach while easing the computational difficulties. We take several steps to investigate whether our model provides better estimates than models currently utilized for responses violating LID. We utilized Markov Chain Monte Carlo (MCMC) simulation to generate responses to a hypothetical test and fit the models to the synthetic data. Next, we test our model using the real Part III responses and compare the obtained IRT parameter estimates and ability levels to the generated ones. (This is a conference presentation abstract and not a full work that has been published.)Adrian Hunnisett, Christina CunliffeObjectives: The aim of this study was to explore student engagement, preferences and perceptions in the delivery of early academic subjects, comparing traditional didactic delivery with the flipped classroom model. Methods: Following relevant ethical approvals, students (n=26) were divided into 2 convenience groups based on course pathway. During 2 biomedical science sessions, one group followed a standard classroom delivery (ST) model whilst the second group followed a flipped classroom (FC) model, covering a topic at home prior to a class tutorial session. Knowledge acquisition was assessed by a collective MCQ at the end of the session (usual practice) and students completed a small survey exploring opinions on the teaching model used. At the second session, the groups were reversed. Results: No significant difference was demonstrated in knowledge acquisition between the 2 groups (p=0.85). Student satisfaction was slightly higher in the FC group (p=0.062), although not quite reaching statistical significance. The reason cited for increased satisfaction was exploration of relevant clinically based case studies to apply the knowledge. Also, motivation to attend sessions was increased in the FC group. Conclusions: This small scale pilot study suggests the FC approach has no benefit over ST in terms of knowledge acquisition. However, there do appear to be some benefits for the student experience that may improve attendance rates. Further research over longer periods and with larger numbers is warranted. (This is a conference presentation abstract and not a full work that has been published.)Kelley Humphries, David ParishIntroduction: Outcome assessments developed for individuals with disabilities are based on assessments for the general population. Presumptions of applicability and irrelevant questions result in unreliable and invalid data. The PROMIS Global Health assessment (PROMIS) is widely accepted and validated for the general population but there are no large studies that that focus on its use for individuals with disabilities in a community-based clinic. PURPOSE To identify and describe how Logan Human Performance Center (HPC) at Paraquad patients respond to each domain of the assessment throughout individualized treatment plans. Introduce research to identify trends for this population in this unique setting. Methods: This study reviewed 40 patient records from HPC. Cross-sectional data analysis looked at the correlation between PROMIS scores and basic demographic information. Results: Individuals who presented with a history of strokes reported the most improvement in their scores from baseline with the most substantial improvement in their raw score for physical health. Fatigue showed the greatest improvement from baseline in all groups (SCI, stroke, MS, surgical impairments, TBI, VI, and other). Patients over the age of 70 showed the most overall improvement in scores across all domains. Raw physical and pain scores showed the greatest global improvement among all age groups. The largest improvement in scores were raw physical in the 60-69 age group. Males showed the most improvement in fatigue scores. Females showed the most improvement in physical health scores. Conclusion: This pilot study, with inherent limitations, helps support the need for quality research in this area. If an outcome assessment like PROMIS can be generalized to accurately measure overall health and well-being in this population, we can better understand what methods across various fields of health care are most effective in promoting health and wellness for individuals with disabilities. (This is a conference presentation abstract and not a full work that has been published.)John Hyland, Igor HimelfarbObjective: OSCEs (Objective Structured Clinical Examinations) are commonly used in the education of healthcare professionals to assess their performance during patient encounters. During high-stakes examinations for regulatory purposes, trained “patients” and examiner checklists are supplemented by selected response questions to assess clinical decision-making and provide standardization. This complexity makes the usual methods of setting a passing standard (the “cut score”) and equating the results quite unwieldy. The U.S. National Board of Chiropractic Examiners (NBCE) sought to update its Part IV practical examination using Item Response Theory (IRT) and a novel standard-setting approach (Bookmark method). Methods: The Part IV examination item responses were calibrated using General Partial Credit IRT models. The models were fitted within domains of the test. All encounter stations and selected response questions from each of the three domains (Diagnostic Imaging, Case Management, and Chiropractic Technique) were arranged in ascending order of difficulty in printed booklets. A regionally representative judging committee comprising 18 chiropractic clinical faculty members was convened. Following instructions and discussion, members worked individually to identify a cut point, beyond which the stations became too difficult to expect a borderline test-taker to proceed successfully. Results: The three domains each required 3 rounds of judgments to come to an acceptable consensus. The entire process was completed in approximately 7 hours. Conclusion: This approach to standard setting and equating proved to be considerably less time-consuming and much easier on the judges than the previous method, which used Classical Test Theory (CTT) and a modified Angoff determination. (This is a conference presentation abstract and not a full work that has been published.)Deborah Kopansky-Giles, Lindsay Beavers, Norman Dewhurst, Lori Whelan, Fok-Han Leung, Ashley Skiffington, Beck McNeilBackground: In today's healthcare environment health professionals need to work effectively in teams to improve quality of care. At St. Michael's Hospital, an interprofessional collaboration competency framework (IPCCF) was developed to improve individual collaborative competencies and support team-based care. Knowledge transfer about the IPCCF occurred through multiple strategies to facilitate competency awareness and attainment amongst clinical staff. Methods: The IPCCF was embedded in the employee performance appraisal (PA) process and due for their PA and their managers were included. Additional education consisted of an IPCCF toolkit and information on the process for staff. Both groups completed modified PA forms and a PA meeting and a post-PA survey was sent to staff. Managers completed a key informant interview, analyzed for themes. Results: An ICU and two medical/surgical units were evaluated. 46 staff participated, 38 (82%) completed the survey. All reported they understood the collaborator competencies and that completing the process with the aid of the toolkit was ‘easy.' Staff self-identified moderate to high confidence in all 6 IPCCF domains. Managers had a good understanding of the IPC competencies and recognized the importance of embedding these in the PA. Manager interviews identified two additional themes: variation in, and the need for, institutional support for the new PA process. Conclusions: Staff successfully completed the modified PA process demonstrating knowledge about the IPCCF concepts and had high self-rated confidence in the IPCCF domains. Managers highlighted the importance of the practice context and of institutional support for the PA process as essential elements in the hospital-wide roll-out of this initiative. (This is a conference presentation abstract and not a full work that has been published.)Nadège Lemeunier, Hainan Yu, Pierre CôtéObjective: Chiropractic students invest their time and effort in physical examination to gain diagnostic skills. However, most of these tests have poor or questionable reliability or validity. We aim to discuss the implications of teaching these tests to chiropractic students and offer a pedagogical solution to this problem. Method: We conducted 20 systematic reviews on the reliability and validity of diagnostic tests used to assess neck and low back pain patients. We synthesize the evidence using best-evidence synthesis. A multidisciplinary panel of 18 clinical and scientific experts used the evidence to develop recommendations about the use of these tests. Only, tests and tools with adequate validity and reliability informed the development of recommendations. Results: Most clinical tests used by chiropractors to assess neck and low back pain patients have poor validity and reliability. Yet, these tests are taught in chiropractic schools. We recommend that educators and clinicians focus on teaching the use of test that have adequate reliability and validity. Educators and clinicians need to understand the impact of diagnostic misclassification and its impact on patient care. Conclusion: Our results will help to standardize the diagnostic pedagogic approach to teach diagnostic tests used for patients with neck and low back pain. (This is a conferenc

Item response theory analysis of Centers for Disease Control and Prevention Health-Related Quality of Life (CDC HRQOL) items in adults with arthritis.

BackgroundExamine the feasibility of performing an item response theory (IRT) analysis on two of the Centers for Disease Control and Prevention health-related quality of life (CDC HRQOL) modules – the 4-item Healthy Days Core Module (HDCM) and the 5-item Healthy days Symptoms Module (HDSM). Previous principal components analyses confirm that the two scales both assess a mix of mental (CDC-MH) and physical health (CDC-PH). The purpose is to conduct item response theory (IRT) analysis on the CDC-MH and CDC-PH scales separately.Methods2182 patients with self-reported or physician-diagnosed arthritis completed a cross-sectional survey including HDCM and HDSM items. Besides global health, the other 8 items ask the number of days that some statement was true; we chose to recode the data into 8 categories based on observed clustering. The IRT assumptions were assessed using confirmatory factor analysis and the data could be modeled using an unidimensional IRT model. The graded response model was used for IRT analyses and CDC-MH and CDC-PH scales were analyzed separately in flexMIRT.ResultsThe IRT parameter estimates for the five-item CDC-PH all appeared reasonable. The three-item CDC-MH did not have reasonable parameter estimates.ConclusionsThe CDC-PH scale is amenable to IRT analysis but the existing The CDC-MH scale is not. We suggest either using the 4-item Healthy Days Core Module (HDCM) and the 5-item Healthy days Symptoms Module (HDSM) as they currently stand or the CDC-PH scale alone if the primary goal is to measure physical health related HRQOL.

Rasch Model Parameter Estimation in the Presence of a Nonnormal Latent Trait Using a Nonparametric Bayesian Approach.

Standard approaches for estimating item response theory (IRT) model parameters generally work under the assumption that the latent trait being measured by a set of items follows the normal distribution. Estimation of IRT parameters in the presence of nonnormal latent traits has been shown to generate biased person and item parameter estimates. A number of methods, including Ramsay curve item response theory, have been developed to reduce such bias, and have been shown to work well for relatively large samples and long assessments. An alternative approach to the nonnormal latent trait and IRT parameter estimation problem, nonparametric Bayesian estimation approach, has recently been introduced into the literature. Very early work with this method has shown that it could be an excellent option for use when fitting the Rasch model when assumptions cannot be made about the distribution of the model parameters. The current simulation study was designed to extend research in this area by expanding the simulation conditions under which it is examined and to compare the nonparametric Bayesian estimation approach to the Ramsay curve item response theory, marginal maximum likelihood, maximum a posteriori, and the Bayesian Markov chain Monte Carlo estimation method. Results of the current study support that the nonparametric Bayesian estimation approach may be a preferred option when fitting a Rasch model in the presence of nonnormal latent traits and item difficulties, as it proved to be most accurate in virtually all scenarios that were simulated in this study.

IRT Parameter Estimation With Response Times as Collateral Information

Hierarchical modeling of responses and response times on test items facilitates the use of response times as collateral information in the estimation of the response parameters. In addition to the regular information in the response data, two sources of collateral information are identified: (a) the joint information in the responses and the response times summarized in the estimates of the second-level parameters and (b) the information in the posterior distribution of the response parameters given the response times. The latter is shown to be a natural empirical prior distribution for the estimation of the response parameters. Unlike traditional hierarchical item response theory (IRT) modeling, where the gain in estimation accuracy is typically paid for by an increase in bias, use of this posterior predictive distribution improves both the accuracy and the bias of IRT parameter estimates. In an empirical study, the improvements are demonstrated for the estimation of the person and item parameters in a three-parameter response model.

Home fruit, juice, and vegetable pantry management and availability scales: A validation

Home fruit, 100% juice, and vegetables (FJV) availability is related to increased FJV consumption by children. While FJV must be purchased for use in the home, no scales have been reported on home FJV pantry management practices. A scale for home FJV pantry management practices was generated from focus group discussions with diverse 162 food shoppers. A commonly used scale of home FJV availability was also assessed. A grocery store intercept survey recruited 171 food shoppers with children in front of supermarkets and grocery stores. Survey instruments were administered twice, separated by 6 weeks. Single dimensionality was observed for each scale. Item Response Theory parameter estimates revealed easily interpreted patterns in the sequence of items by difficulty of response. These scales are available to help better understand influences on family FJV purchase decisions.

Parent outcome expectancies for purchasing fruit and vegetables: a validation

To validate four scales -- outcome expectancies for purchasing fruit and for purchasing vegetables, and comparative outcome expectancies for purchasing fresh fruit and for purchasing fresh vegetables versus other forms of fruit and vegetables (F&V). Survey instruments were administered twice, separated by 6 weeks. Recruited in front of supermarkets and grocery stores; interviews conducted by telephone. One hundred and sixty-one food shoppers with children (18 years or younger). Single dimension scales were specified for fruit and for vegetable purchasing outcome expectancies, and for comparative (fresh vs. other) fruit and vegetable purchasing outcome expectancies. Item Response Theory parameter estimates revealed easily interpreted patterns in the sequence of items by difficulty of response. Fruit and vegetable purchasing and fresh fruit comparative purchasing outcome expectancy scales were significantly correlated with home F&V availability, after controlling for social desirability of response. Comparative fresh vegetable outcome expectancy scale was significantly bivariately correlated with home vegetable availability, but not after controlling for social desirability. These scales are available to help better understand family F&V purchasing decisions.

The factor structure of test task characteristics and examinee performance

The present study focuses on the task characteristics of reading passages and key sentences in a test of second language reading. Using a new methodological approach to describe variation in test task characteristics and explore how differences in these characteristics might relate to examinee performance, it posed the two following research questions: First, how do the characteristics of the texts used in a high-stakes test of English for Academic Purposes reading vary? Second, what relationships exist between its text characteristics and examinee performance? An expanded test task characteristics instrument was constructed, following Freedle and Kostin (1993) and Bachman et al. (1996), and adding a large number of syntactic features (Celce-Murcia and Larsen-Freeman, 1999). Ratings and numerical counts were compiled for three forms of the Test of English as a Foreign Language (TOEFL) Reading Comprehension Section. Taking items as the object of measurement, the results were then used in a series of exploratory and confirmatory factor analyses, along with IRT (item response theory) parameter estimates for the items in question.

DIF Identification Using HGLM for Polytomous Items

The relationship between the hierarchical generalized linear model (HGLM) and item response theory (IRT) models has been demonstrated for dichotomous items. The current study demonstrated the use of the HGLM for polytomous items (termed PHGLM) for identification of differential item functioning (DIF). First, the algebraic equivalence between parameterizations of a constrained form of Muraki's rating scale model and PHGLM was demonstrated. Next, as a precursor to using the PHGLM for DIF identification, a simulation was conducted to compare estimation of item and person latent trait parameters using HLM 5 and PARSCALE software. The resulting PHGLM and IRT parameter estimates were comparable. Last, a simulation was conducted that compared the performance of PHGLM with that of the generalized Mantel-Haenszel statistic for DIF identification. Both approaches performed similarly. Benefits and limitations of using PHGLM for DIF identification are presented. Index terms: hierarchical linear modeling, item response theory, polytomous items, parameter estimation, differential item functioning, attitude measurement.

A Comparison of Bootstrap Standard Errors of IRT Equating Methods for the Common-Item Nonequivalent Groups Design

The primary purpose of this study was to compare bootstrap standard errors of 5 item response theory (IRT) equating methods for the common-item nonequivalent groups design. For true-score (Method 1) and observed-score (Method 2) equating, IRT parameters were estimated separately, and a linear scaling transformation method was used to rescale the IRT parameter estimates for Form X onto the Form Y scale. For IRT chained true-score equating (Method 3), IRT parameters for Form X and Form Y were estimated separately, and then IRT chained true-score equating was performed. For the last 2 methods, IRT parameters for both forms were estimated simultaneously. Using the simultaneously estimated parameter estimates, IRT true-score (Method 4) and observed-score (Method 5) equatings were performed. For each method, the standard deviation was computed over 500 bootstrap replications to obtain the standard error of IRT equating at each raw score point for the new form. The estimated bootstrap standard errors for Methods 4 and 5 were slightly less than those for Methods 1 and 2. Method 3 produced the greatest standard errors. However, the standard errors for all 5 methods were small enough to suggest that standard errors of equating less than 0.1 standard deviation units could be obtained with any method, even with sample sizes of 500.

FIELD TEST OF A COMPUTER‐BASED GRE GENERAL TEST

ETS Research Report SeriesVolume 1993, Issue 1 p. i-47 ArticleFree Access FIELD TEST OF A COMPUTER-BASED GRE GENERAL TEST Gary A. Schaeffer, Gary A. SchaefferSearch for more papers by this authorClyde M. Reese, Clyde M. ReeseSearch for more papers by this authorManfred Steffen, Manfred SteffenSearch for more papers by this authorRobert L. McKinley, Robert L. McKinleySearch for more papers by this authorCraig N. Mills, Craig N. MillsSearch for more papers by this author Gary A. Schaeffer, Gary A. SchaefferSearch for more papers by this authorClyde M. Reese, Clyde M. ReeseSearch for more papers by this authorManfred Steffen, Manfred SteffenSearch for more papers by this authorRobert L. McKinley, Robert L. McKinleySearch for more papers by this authorCraig N. Mills, Craig N. MillsSearch for more papers by this author First published: June 1993 https://doi.org/10.1002/j.2333-8504.1993.tb01518.xCitations: 23AboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Citing Literature Volume1993, Issue1June 1993Pages i-47 ReferencesRelatedInformation

A Comparative Study of the Effects of Recency of Instruction on the Stability of IRT and Conventional Item Parameter Estimates

A potential concern for individuals interested in using item response theory (IRT) with achievement test data is that such tests have been specifically designed to measure content areas related to course curriculum and students taking the tests at different points in their coursework may not constitute samples from the same population. In this study, data were obtained from three administrations of two forms of a Biology achievement test. Data from the newer of the two forms were collected at a spring administration, made up of high school sophomores just completing the Biology course, and at a fall administration, made up mostly of seniors who completed their instruction in the course from 6–18 months prior to the test administration. Data from the older form, already on scale, were collected at only a fall administration, where the sample was comparable to the newer form fall sample. IRT and conventional item difficulty parameter estimates for the common items across the two forms were compared for each of the two form/sample combinations. In addition, conventional and IRT score equatings were performed between the new and old forms for each o f the form sample combinations. Widely disparate results were obtained between the equatings based on the two form/sample combinations. Conclusions are drawn about the use o f both classical test theory and IRT in situations such as that studied, and implications o f the results for achievement test validity are also discussed

Putting Item Response Theory into Perspective

Abstract Item response theory (IRT) is a mathematically rigorous and computationally complex measurement theory which has potential utility in psychomotor measurement. The present paper complements Spray's heuristic description of IRT and one of its applications “adaptive testing“; presenting additional applications of IRT in physical education, drawbacks of IRT parameter estimation and theory, and recommendations for future IRT research in physical education.